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Background 


• INSIDE has lots of geometric detail, interleaved layers of transparency 

• camera always slightly moving => lots of crawling 

• ... wanted clean, stable images 

• began looking into temporal AA early 201 4 


• quickly became primary AA solution 


Temporal Anti-Aliasing? 


spatio-temporal post-process technique (... what?) 


correlates new fragments with fragments from history buffer 


output becomes next frame in history (feedback loop) 


sub-pixel information recovered over time 





What it looks like 


no AA 


our temporal AA 



What it looks like ... 



no AA 


our temporal AA 



First some basic intuition 


• local region of a surface fragment may 
remain in view across multiple frames 

• if relationship between viewer and 
subject changes every frame, then 
rasterization => variation 

• if we step back in time, then we 
can use this variation to refine the 
current frame 


view N 



Stepping back in time 


• want to correlate current frame 
fragments with fragments from 
previous frame(s) 

• can do spatially, with reprojection 

o relies on depth buffer information 
o limited to closest written fragment 

• not always possible 

o sometimes the data just isn’t there 




Stepping into void 


• fragments can become occluded or 
disoccluded at any time, making it 
difficult to accurately step back 

o bummer., but let’s get back to that later 

• if relationship between viewer and 
subject never changes, there is no 
additional information to be gained 
from stepping back... 




Step 1 : Jitter your view frustum 


• have established that if camera is static, 
then we are losing information 

• thus, every frame, prior to rendering: 

o get texel offset from sample distribution 
o use offset to calculate projection offset 
o use projection offset to shear frustum 



... more on sample distribution later 



Step 2: For every fragment 
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Reprojection of static scenes 


• start in current fragment p_uv 


raster 



Reprojection of static scenes 


• start in current fragment p_uv 




Reprojection of static scenes 


• start in current fragment p_uv 

• reconstruct world space p using depth 
and frustum params for current frame 

o lerp corner ray, scale by linear depth 




Reprojection of static scenes 


• start in current fragment p_uv 

• reconstruct world space p using depth 
and frustum params for current frame 

o lerp corner ray, scale by linear depth 

• then, reproject p into previous frame 

° q_cs = mul( VP_prev’, p ) 

° q_uv = 0.5 * ( q_cs.xy / qcs.w ) + 0.5 




Reprojection of static scenes 


• start in current fragment p_uv 

• reconstruct world space p using depth 
and frustum params for current frame 

o lerp corner ray, scale by linear depth 

• then, reproject p into previous frame 

° q_cs = mul( VP_prev’, p ) 

° q_uv = 0.5 * ( q_cs.xy / qcs.w ) + 0.5 

• history sample is then 

o c_hist = sample( buf_history, q_uv ) 




Reprojection of dynamic scenes 


• for dynamic scenes we need a velocity buffer 

o separate pass before temporal 
o initialize to camera motion using static reprojection 

■ v = p_uv - q_uv 

o then render dynamic objects on top 

■ v = compute_ssvel( p, q, VP, VP_prev’ ) 

• reprojection step becomes read and subtract 

o v = sample( buf_velocity, p_uv ) 

° q_uv = p_uv - v 



buf_history 





Reprojection and edge motion 


• should add: we don’t actually sample v directly in p_uv 

o else out-of-edge fragments will not travel with occluder 

• using velocity of closest (depth) fragment within 3x3 region 

o v = sample( buf_velocity, closest_fragment( p_uv ).xy ) 

• similar to suggestion by [Karisl 4] 


• result: nicer edges in motion 



Reprojection and edge motion ... 




v from same fragment 


v from closest fragment 3x3 


Revisiting overview ... 
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Constraining history sample 


• history sample sometimes invalid 

o because of occlusion / disocclusion 
o because reprojection tracks only opaque 
o ( . . . and we have lots of transparency ) 

• what if we trivially accept? 

o ghosting / smearing 
o example on the right 


• have to constrain 


Constraining history sample ... 


• depth based rejection, velocity weighing [Sousall] [Jimenezll] 


• attempted this, found too fragile for our case 

o hard to eliminate ghosting with sliding threshold 
o ( ... in history, threshold itself is ghosting ) 

• also: transparency layers still smearing 

o didn’t want to run temporal after opaque! 
o needed something else, so back to the brick wall 



• neighbourhood clamping to the rescue. 


view N 



Neighbourhood clamping 101 


• [Sousal 3] clamp history to neighbourhood of current sample 


o essentially per-frame upper bound on reprojection error 
o clamp color to min-max of 4 taps and center texel 
o big improvement in stability over velocity weighing 


• pure color space operation 

o cn_min = sampleJocal_min( buf_color, p_uv ) 
o cn_max = ...II similar 

o c hist’ = clamp( c_hist, cn_min, cn_max ) c 

A 



c hist 


r 



c hist’ 


Neighbourhood clamping, first pass 

• during production, the first implementation was a 
dynamic variation of the 4-tap approach 

o variable distance to 4 sample points, decided per-pixel 
o higher velocity => closer to center texel (strict on motion) 
o decent results without requiring per-object velocities 

• we used this for about a year(!) 

o “early” first pass enabled artists to tailor effects and content 

• later. . . decided to add per-object velocities 

o axed dynamic 4-tap approach in favor of image quality 
o switched to rounded 3x3 neighbourhood and clipping 


-V 



sample offset 0.5-0.666 
from texel center 




Neighbourhood clamping, now clipping 


• [Karis14] larger “rounded” neighbourhood, clip > clamp 

o min-max of 3x3 neighbourhood 

o blend with min-max of 5 taps in “+’ pattern - 

o bit more expensive, but better image quality 


• clipping prevents clustering when colorspace 
is distant from history sample 





A little note on line-box clipping 


• proper line clip is “slow” 

• we just clip towards aabb center 

o transform color vector into unit space 
o calc divisor and apply in clip space 


// note: clips towards aabb center + p.w 
float4 clip_aabb( 


f loat3 

aabb 

min. 

// 

cn min 



f loat3 

aabb 

max, 

// 

cn max 



f loat4 

Pf 


// 

c in' 



f loat4 

q) 


// 

c hist 



float3 p 

clip 

= 0. 

5 * 

(aabb max 

+ aabb 

min) ; 

float3 e 

clip 

= 0. 

5 * 

(aabb max 

- aabb 

min) ; 


cn max 



c hist 


float4 v clip 
float3 v unit 
float3 a_unit 
float ma unit 


q - float4 (p_clip, p.w); 
v clip.xyz / e clip; 
abs (v_unit) ; 

max(a unit.x, ax (a unit.y, 
a unit . z) ) ; 


if (ma unit > 1.0) 

return float4 (p_clip, p.w) + v clip / ma unit; 
else 

return q;// point inside aabb 


Revisiting overview ... 













Final blend, weighing constrained history 


• weigh constrained history and unjittered input 

o c hist’ = ...II constrained history sample 
o c_in’ = sample( buf_color, unjitter( p_uv ).xy ) 
o c_feedback = lerp( c_in’, c_hist’, k_feedback ) 

• update history buffer and copy to output 

o rt_history = c_feedback 

o rt_output = blit( rt_history ) 

• want to use high feedback factor to increase retention 

o beware of artefacts 



Trailing artefacts 


• history fragments can linger if none of their 
neighbours force them out 

• observation: boy silhouette fragments 

o fast motion during turns, landings, etc. 

• only distinct at artificially low resolution and 
framerate, wanted to remedy anyway 

• idea : conceal with output-only motion blur 

o target history and output in same pass with MRT 



Big picture 2.0: Adding motion blur to the mix . . . 














Final blend with motion blur fallback 


• update history buffer just like before 

o rt_history = c_feedback 

• for output target, blend with motion blurred input 

o c_motion = sample_motion( buf_color, unjitter( p_uv ), v ) 

o rt_output = lerp( c_motion, c_feedback, k_trust ) 

o k_trust = invlerp( 15, 2, ||v|| )// works well for us. 


k trust 



• forces transition to motion blur (no history!) for fast moving fragments 

o includes immediate neighbours, due to v relying on closest_fragment( ... ) 


Final blend with motion blur fallback ... 





On picking a good sample distribution 


• lots of trial and error, took practical approach 

• ... head close to screen, magnifying glass, 
obsessing over high contrast regions 

• wanted to find good balance between quality 
and speed of convergence 


• heuristics: side-scrolling game 



On picking a good sample distribution 



... inspecting many pixels 


On picking a good sample distribution ... 


• using exponential history 

o samples weigh less over time 

o need high feedback factor 

■ avoid visible cycle 


• nice to revisit same sub-pixel regions often 



0.185 


► 

1 6 frames 


o clamp/clip will compress tail 

o quickly return to that data 

• initially used very few sample points 



Some of the sequences tested 



halton(2,3) x8 


halton(2,3) x16 











Closing remarks on sample distributions 


• while using 4-tap neighbourhood, “uniform 4 helix” was my favourite 

o short cycle => when sample is rejected, comes back to it quickly 
o not regular uniform 4 

■ every step crosses horizontal center line 

■ good at closing horizontal seams 

• after moving to 3x3 and clipping, switched to 16 indices of halton(2,3) 

o much better coverage => much nicer edges 
o revisits sub-pixel regions quickly despite cycle length 

• thought about motion-perpendicular pattern; needs more cooking time 

perhaps squeeze along line of camera motion? 


o 



Summary of implementation 


• jittering view frustum 

o 1 6 first samples of halton(2,3) 

• generating velocity buffer 

o camera motion + dynamics (manual tagging, eurgh) 

• reprojection using velocity 

o based on closest (depth) fragment 

• neighbourhood clipping 

o center-clip to RGB min-max of “rounded” 3x3 region 

• motion blur fallback 

o kicks in when ||v|| > 2, and full effect at 1 5 
o does not apply to history 


temporal pass 
~1 .7ms on xbl 
@ 1920x1080 



Was greatly inspired by 


• [Yang09] individual sub-pixel buffers, reprojection 

( Amortized Supersamplinq ) 

• [Sousal 1 ] [Jimenezl 1 ] exponential history, velocity weighing 

( Anti-Aliasing Methods in CryENGINE 3 ) 

• [Sousal 3] neighbourhood clamping; “SMAA-ltx” 

( CryENGINE 3 Graphics Gems ) 

• [Karisl 4] clipping over clamping, YCoCg constraints 

( High Quality Temporal Supersamplinq ) 

• [McGuire12] motion blur reconstruction filter 

( A Reconstruction Filter for Plausible Motion Blur ) 


Temporal also has some 


• stochastic everything 

o shadows 
o reflections 
o volumetries 


discussed as part of talk 
about INSIDE rendering :) 
definitely go see it. 


ally nice side-effects™ 




job@playdead.com 


That’s it! Thank you for coming. 


Questions? 


full source code: https://qithub.com/plavdeadqames/temporal/ 
email me at lasse@playdead.com 
* @codeverses 


Bonus slides 



Clipping in YCoCg 


• [Karis14] suggests clipping in YCoCg instead of RGB 

• Intel has a nice page with illustrations and the transformations 

• ... ultimately not used for INSIDE 

• our implementation still supports it 


